Web scraping LinkedIn means collecting the public data LinkedIn serves to logged-out visitors: profiles, company pages, job posts, and job search results. LinkedIn serves those surfaces to a point, then asks for a sign-in once one address requests too much too fast. Reliable collection at volume depends on the right proxy type and sensible request pacing, with a clear view of LinkedIn's terms and the law that applies to you.
Key takeaways
The right proxy depends on the LinkedIn surface. Public job pages and job search answer logged-out requests readily, so datacenter, ISP, and unlimited bandwidth proxies deliver the throughput they need. Public profiles and company pages are rate-limited per session, so rotating residential proxies that route through real consumer networks in your target markets fit collection at scale. For the highest-volume, longest-running jobs, rotating mobile proxies add the widest IP diversity.
Most guides on this topic skip the proxy question, because they sell a finished scraper with the proxy layer hidden inside. If you build your own collector, it's the first decision you make, and it shapes your success rate as much as your bill.
| LinkedIn surface | Access behaviour | Recommended proxy type | Why it fits |
|---|---|---|---|
| Public job pages and job search | Answers logged-out requests readily | Datacenter IPv4, ISP, or unlimited bandwidth | High throughput and stable sessions for large volumes of job data |
| Public profiles | Served logged-out up to a per-session volume, then a sign-in prompt | Rotating residential | A fresh consumer IP per request or session, in the market you collect from |
| Company pages | Overview served logged-out; employee lists need a sign-in | Rotating residential | Same per-session pattern as profiles, so the same rotation logic fits |
| High-volume sustained collection | Long runs across many surfaces and markets | Rotating mobile | Carrier networks with broad IP diversity for the largest jobs |
| Fixed-IP or allowlist need | One stable address over days or weeks | Static residential (ISP) | Fixed residential IPs with long session continuity |
Cost is the other half of the decision. Most buyers think in cost per 1,000 successful requests, and that's the right lens. At Proxy-Cheap, rotating residential and rotating mobile are billed per GB with no monthly commitment, so you can test one surface with a small top-up before you commit to a plan.
Setup has three choices: proxy type, session mode, and pacing. Pick the proxy type for your surface, then choose rotating sessions for broad coverage or sticky sessions when a task needs the same IP for several requests. Authenticate by username and password or by IP allowlist, and pace requests to sit within normal browsing volumes so sessions stay stable.
Generate your credentials in the Proxy-Cheap dashboard first. Each proxy comes with a host and port plus login details. If you're new to the options, our guide to proxy types explains the differences in more depth.
Here's a minimal Python setup with httpx, tested on version 0.28.1:
python import httpx PROXY = "http://USERNAME:PASSWORD@HOST:PORT" # copy from your Proxy-Cheap dashboard HEADERS = { "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 " "(KHTML, like Gecko) Chrome/141.0.0.0 Safari/537.36", "Accept-Language": "en-US,en;q=0.9", } with httpx.Client(proxy=PROXY, headers=HEADERS, timeout=20) as client: r = client.get( "https://www.linkedin.com/jobs/search", params={"keywords": "data engineer", "location": "Germany"}, ) print(r.status_code) # 200
If you prefer requests, pass proxies={"http": PROXY, "https": PROXY} to requests.get() instead. For a SOCKS5 line, install httpx[socks] and start the proxy URL with socks5://.
The same request in cURL is a quick way to check credentials from the terminal:
bash curl -x "http://HOST:PORT" -U "USERNAME:PASSWORD" \ "https://www.linkedin.com/jobs/search?keywords=data%20engineer&location=Germany" \ -o jobs.html -w "%{http_code}\n"
A 200 means the proxy and the page both answered. A 407 means the proxy rejected your login details.
A rotating session gives you a new IP on every request, which suits breadth, like a list of 5,000 job URLs. A sticky session keeps one IP for a while, for tasks that need several requests in a row from the same address, such as reading a profile and then its public activity. In the Proxy-Cheap credentials generator, "Random IP" rotates per request and "Session IP" holds one address for about 30 minutes.
Rotating residential proxies authenticate with a username and password only. Static residential and datacenter proxies support both, so you can allowlist your server's IP and keep credentials out of your code. If you manage proxies in the browser instead of a script, the Chrome proxy extension handles switching and rotation in a couple of clicks. To create proxies or rotate IPs from code, see the Proxy-Cheap API documentation.
LinkedIn doesn't publish a request rate for logged-out visitors, so you read the limit from its responses. A redirect to a sign-in page or an HTTP 429 (too many requests) means you've gone past it, and the right move is to slow down. This helper waits a few seconds between pages and stops cleanly when LinkedIn asks for a pause:
python import random import time def fetch(client, url, params=None): """Return the response, or None when LinkedIn asks you to slow down or sign in.""" r = client.get(url, params=params) if r.status_code == 429 or "authwall" in r.headers.get("location", ""): return None time.sleep(random.uniform(3, 7)) # a few seconds between pages return r
httpx doesn't follow redirects by default, so the sign-in redirect shows up in the Location header where you can see it. Log every None, back off, and resume later at a lower rate.
Public LinkedIn surfaces expose a defined set of fields. Profiles show name, location, current role and organization, and education. Company pages add industry, headcount, headquarters, and specialties. Public job pages and job search show title, company, location, posting date, employment type, and description. None include private contact details or member-only analytics, which require a sign-in.
Coverage on profiles varies from person to person, because members choose how much of their profile is public. Company pages also list the company type and founding year, while the employee list asks for a sign-in. A single job posting adds seniority level to the card fields.
Job search cards cover most hiring-trend work, so job data is the usual starting point. If your project feeds a market research pipeline, job cards alone can track hiring by role and city over time.
Some data is never public. Contact details, connection lists, and profile view counts all sit behind a login. People search asks for a sign-in even for logged-out visitors, so it falls outside public collection. For a wider look at the workload side, see our web data collection use case.
LinkedIn delivers public fields three ways. Some fields sit in the server-rendered HTML and can be read from the response directly. Others load inside script tags as JSON-LD, so the parser reads that script tag. Job search and company job lists come from a guest background endpoint that returns results in pages as you scroll.
Job search result cards and company overview sections arrive as plain HTML in the first response. You read them with CSS selectors or XPath. This parser pulls job cards with BeautifulSoup and lxml:
python from bs4 import BeautifulSoup def parse_cards(markup): soup = BeautifulSoup(markup, "lxml") jobs = [] for card in soup.select("div.base-card"): def text(selector): node = card.select_one(selector) return node.get_text(strip=True) if node else None link = card.select_one("a.base-card__full-link") posted = card.select_one("time") jobs.append({ "title": text("h3.base-search-card__title"), "company": text("h4.base-search-card__subtitle"), "location": text("span.job-search-card__location"), "posted": posted["datetime"] if posted else None, "url": link["href"].split("?")[0] if link else None, }) return jobs
Most public pages also carry structured data in JSON-LD, the W3C format for linked data. Load each application/ld+json script tag as JSON, then pick the item you need by its @type. Profiles use Person, company pages use Organization, and job pages use JobPosting.
python import json from bs4 import BeautifulSoup def read_json_ld(markup): soup = BeautifulSoup(markup, "lxml") items = [] for tag in soup.select('script[type="application/ld+json"]'): data = json.loads(tag.string) items.extend(data.get("@graph", [data])) # profiles nest items in @graph return items job = next(i for i in read_json_ld(r.text) if i.get("@type") == "JobPosting") print(job["title"], job["datePosted"], job["employmentType"])
The job description field is HTML-encoded, so run it through html.unescape() and BeautifulSoup to get clean text. JSON-LD tends to outlast CSS class names, because it follows the public schema.org vocabulary rather than LinkedIn's own markup.
When you scroll a logged-out job search, the page calls a guest endpoint for the next batch: https://www.linkedin.com/jobs-guest/jobs/api/seeMoreJobPostings/search. It takes keywords, location, and a start offset, and returns job cards as HTML fragments. You can spot calls like this yourself by scrolling the page in a headless browser while you watch its network activity. Then call the endpoint directly:
python GUEST_SEARCH = "https://www.linkedin.com/jobs-guest/jobs/api/seeMoreJobPostings/search" all_jobs, start = [], 0 with httpx.Client(proxy=PROXY, headers=HEADERS, timeout=20) as client: while start < 100: r = fetch(client, GUEST_SEARCH, {"keywords": "data engineer", "location": "Germany", "start": start}) if r is None: break # LinkedIn asked for a pause, so resume later batch = parse_cards(r.text) if not batch: break all_jobs.extend(batch) start += len(batch) print(len(all_jobs))
The loop steps start based on how many cards come back, so it doesn't depend on a fixed page size. Selectors and endpoints change without notice, so re-test your parser before each run. A parser that silently returns empty fields does more damage than one that fails loudly. To manage proxies through code as well, read our explainer on the proxy API.
Collecting public LinkedIn data sits in a nuanced legal position. In hiQ Labs v. LinkedIn, the Ninth Circuit found that reading public pages does not by itself breach the Computer Fraud and Abuse Act. That ruling did not make every project lawful: LinkedIn's User Agreement still forbids automated collection, and copyright, contract, and data-protection law such as GDPR apply. Collect only public data, honor the terms and applicable law, and consult a lawyer for your case.
The case is often quoted as blanket permission, so the details matter. hiQ Labs v. LinkedIn started in 2017, after LinkedIn sent hiQ a cease-and-desist letter over its collection of public profiles. In 2019, and again in April 2022 after the Supreme Court's Van Buren decision, the Ninth Circuit held that scraping publicly accessible data likely isn't access "without authorization" under the CFAA. Both rulings came at the preliminary injunction stage.
The case didn't end there. In November 2022, the district court found that hiQ had breached LinkedIn's User Agreement through automated collection and through contractors who created fake profiles. The parties settled that December with a consent judgment. hiQ paid $500,000, agreed to stop collecting LinkedIn data, and had to delete the data and code built from it.
| What hiQ v. LinkedIn settled | What it did not settle |
|---|---|
| Reading public pages likely isn't "without authorization" under the CFAA (Ninth Circuit) | Whether collection is allowed under LinkedIn's contract terms |
| A site owner's objection alone doesn't make public pages off-limits under the CFAA | Copyright, data-protection, and privacy claims |
| Rules outside the US, including GDPR |
The contract question is the live one. The LinkedIn User Agreement (section 8.2) says members must not develop, support, or use software, scripts, robots, or other means to scrape or copy the Services, including profiles and other data. LinkedIn's robots.txt carries a similar notice about automated access. That exposure is separate from the CFAA question, and it applies most clearly when you hold a LinkedIn account, because that's when you've agreed to the terms.
Data protection is the other half. Under GDPR, a name tied to a job title is personal data, even when it's public. If you collect data on people in the EU, you need a lawful basis (usually legitimate interest, backed by a written assessment), and you should collect only the fields your purpose needs. Job and company data carry far less risk than profile data, which is one more reason many teams start there.
Proxy-Cheap's own appropriate use policy applies to every workload on our network. This section is general information, not legal advice, so talk to a lawyer about your specific project before collecting personal data.
Public collection usually stalls for three reasons. A single address that requests too many pages too fast triggers a sign-in prompt. Requests from datacenter ranges used for hosting can be treated differently from consumer traffic. And running one long session without variation raises failure rates. Reliable residential and mobile proxies, with sensible pacing and rotation, address all three while keeping your request rate modest.
You can collect public LinkedIn data with open-source libraries, the official API, or no-code workflows. Python libraries such as requests, httpx, and BeautifulSoup handle fetching and parsing. Open-source repositories on GitHub package profile and job logic. LinkedIn's official API covers approved partner use cases. Whichever route you pick, the proxy layer underneath keeps collection reliable at volume.
| Route | Skill level | Control | Best for |
|---|---|---|---|
| Open-source Python | Developer | Full control of requests and parsing | Custom pipelines at volume |
| Official LinkedIn API | Developer plus partner approval | Defined by the partner program | Approved product integrations |
| No-code (n8n and similar) | Operator | Moderate | Small, repeatable monitoring jobs |
Every route needs the same proxy layer underneath. Datacenter proxies with dedicated IPv4 or IPv6 addresses handle daily job crawls at a predictable per-IP price, and IPv4 comes with a 7-day trial. ISP proxies come in static and rotating options for longer sessions. For very large job datasets, unlimited bandwidth proxies give you SOCKS5 access across 195+ locations with no data caps.
Ready to test a LinkedIn collection setup? Start with pay-as-you-go rotating residential or mobile proxies and run the scripts above on one surface. Measure your cost per 1,000 pages before you scale. There are no set-up costs, and you can cancel anytime.
